The present invention relates to remote control of a device, such as an unmanned aerial system (UAS), unmanned aerial vehicle (UAV) or other vehicle or device, and more particularly to a method and system to control operation of a device using an integrated simulation with a time shift option.
One issue with remotely controlling operation of a system or device is latency. This pertains most frequently to system or devices that are at least partly controlled by a remote person or computer. Latency may be defined generally as the period of time between an operator's control input and the operator sensing a system response to the input. Latency tends to limit the rapidity with which control inputs can be made in response to sensor measurements.
Typical components of the latency may include processing the sensor signal; dwell time of the signal waiting for its spot in the data stream; signal time on route from the remote device or system's transmitter to a base station receiver; signal processing by the base station; possible display of the signal; calculation of the desired control response by the base station computer or operator; processing of this response; dwell time of the response waiting for a slot in the data stream; transmission time between base transmitter and the remote receiver; signal processing by the remote system; and control motion delay. In some systems, latency can reach levels that seriously impede system capability. For instance, in the Mars Rover, latency is on the order of about 20 minutes due to the great distance between the Rover on Mars and the base station on Earth. Control of the system by Earth-based operators must be extremely measured and slow in order to make control inputs based on accurate sensor data.
One solution for latency is to reduce latency in each part of the control system chain. However, such reductions may be limited in some situations. In the case of distant operations, the simple delay of signals due to the limited speed of light may be a constraint. Other latency components may also be difficult to eliminate or reduce.
Another solution may be to move more of the control system processing tasks onboard the device or vehicle. In this way, the vehicle is autonomous in its short-term operation and receives controls from a base station less frequently and controls that are more general in nature. However, it can be advantageous to employ human control of some types of systems, especially when the consequences of error are great or when the operating context or environment is complex and uncertain. On the one hand, moving control system processing tasks onboard could mean placing a human operator onboard or within the system. One disadvantage to this arrangement is possible exposure of the operator to potential hazards. Another disadvantage is that provisions for an onboard human operator may increase the complexity, weight and cost of the system. Alternatively, moving increasing control authority from the base station to the remote system with a non-human operator reduces the extent to which the overall system is controllable by a human. One drawback to this is that the system may “decide” to take an action that a human operator might decline to make for one reason or another.
Another possible solution for latency may be operating in a repetitive move-wait cycle. The means of controlling operation is used for some commands for unmanned space vehicles, such as the Mars Rover. Using a slow move-wait cycle may reduce the productivity of the system because the system cannot perform during the “wait” portion of the cycle. Such a method of control may also mean that the system must be designed in such a way that it is stable (if not stationary) during the wait portion of the cycle. This additional constraint on the system may impose penalties on weight, complexity or cost.
Additionally, the operation of some systems involves the potential for operator error. As used herein, “operator” may apply to a human operator, or a non-human or computer operator. For example, for some missions it may be desirable for an unmanned aerial vehicle to fly as close as possible to the ground at a high rate of speed. Such operation is sometimes referred to as “terrain following”. Terrain following becomes difficult over variable or hilly terrain. The minimum altitude at which the vehicle can fly is limited in part by the vehicle's ability to accelerate vertically (up and down) via elevator control inputs. All air vehicles are limited in this regard. Another limitation on altitude is the ability of the remotely located operator or control system to judge the correct point to pull up to avoid a mountain or push down to dive into a valley. A small delay in pulling up may result in an inevitable collision with the mountain. Pushing into a valley slightly too soon may also result in a collision. Further, in ideal, calm or still conditions an optimal path may possibly be calculated in advance of an actual flight. In practice, however, there may be variations in conditions during the actual flight that cannot be foreseen or precisely predicted. These variations may include wind, differences in wind direction, turbulence, updrafts and downdrafts. Further unforeseen conditions may include new obstacles, such as towers, power lines or the other obstacles or hazards. The uncertainty in conditions may be dependent in part on the period of time between the latest measurements or observations and the actual operation or flight, the longer the period, the greater the uncertainty.
One method to reduce operator error is to operate the system with a “margin of safety” that permits the continued safe operation of the system in the event of an operator error or an unforeseen condition. In general, the margin of safety may be statistically determined to reduce errors to an acceptable level. However, a margin of safety generally imposes an operational penalty with respect to one or more measures of merit.
Another method to reduce operator error may be to replace human operators with computers. However, this may not be without its drawbacks. Generally, the software to operate complex systems autonomously is complex and expensive, especially when the system must be very reliable. Additionally, human operators are generally considered to be more flexible and resilient in the face of unforeseen circumstances and more competent to make extremely important decisions under complex circumstances.